Feature point integration positioning system and feature point integration positioning method

ABSTRACT

A feature point integration positioning system includes a moving object, an image input source, an analyzing module and a positioning module. The image input source is disposed at the moving object to shoot an environment for obtaining a sequential image dataset. The analyzing module includes a machine vision detecting unit configured to generate a plurality of first feature points in each of the images based on each of the images, a deep learning detecting unit configured to generate a plurality of second feature points in each of the images based on each of the images, and an integrating unit configured to integrate the first feature points and the second feature points in each of the images into a plurality of integrated feature points in each of the images. The positioning module confirms a position of the moving object relative to the environment at each of the time points.

BACKGROUND Technical Field

The present disclosure relates to a feature point integrationpositioning system and a feature point integration positioning method.More particularly, the present disclosure relates to a feature pointintegration positioning system and a feature point integrationpositioning method applied for the visual SLAM.

Description of Related Art

A simultaneous localization and mapping (SLAM) technique is to detectthe features of the environment during a moving process of an object toconstruct the map of the environment and identify the relation betweenthe object and the environment. With the characteristic that thelocalization and the mapping can be done simultaneously, the requirementof the SLAM is increased and the SLAM is applied for the indoorautomated packing, the warehouse logistics management, and the phoneexhibition tour, etc. Moreover, the visual SLAM, which detects theimages, is more widely used than the radar SLAM, which detects the pointclouds, in the market owing to the cost of the sensors.

For the visual SLAM, the positioning stability is also very important inaddition to the positioning accuracy. Lack of stability is the biggestproblem of the conventional SLAM, which results in position loss duringthe positioning process. Moreover, the conventional SLAM has a problemthat it spends too much time to find the original position after losingthe position, and the position lost problem is obvious in the situationwith severe environment variations, such as corners and locations havinglight contrast. In addition, the conventional SLAM has poor positioningaccuracy outside, and is easily affected by environment variations, suchas locations having high light contrast caused by the front lighting andthe back lighting, road corners and different car arrangements, to losethe position or the mapping.

Based on the abovementioned problems, how to increase the positioningstability of the visual SLAM becomes a pursued target for practitioners.

SUMMARY

According to one aspect of the present disclosure, a feature pointintegration positioning system includes a moving object, an image inputsource, an analyzing module and a positioning module. The image inputsource is disposed at the moving object and configured to shoot anenvironment for obtaining a sequential image dataset. The sequentialimage dataset includes a plurality of images, and each of the imagescorresponds to each of a plurality of time points. The analyzing moduleis signally connected to the image input source to receive thesequential image dataset. The analyzing module includes a machine visiondetecting unit configured to generate a plurality of first featurepoints in each of the images based on each of the images, a deeplearning detecting unit configured to generate a plurality of secondfeature points in each of the images based on each of the images, and anintegrating unit configured to integrate the first feature points andthe second feature points in each of the images into a plurality ofintegrated feature points in each of the images. The positioning moduleis signally connected to the analyzing module, and the positioningmodule receives the integrated feature points in each of the images toconfirm a position of the moving object relative to the environment ateach of the time points.

According to another aspect of the present disclosure, a feature pointintegration positioning method includes a shooting step, an analyzingstep, an integrating step and a positioning step. In the shooting step,an environment is shot by an image input source to obtain a sequentialimage dataset, the sequential image dataset includes a plurality ofimages, and each of the images corresponds to each of a plurality oftime points. In the analyzing step, a plurality of first feature pointsin each of the images are generated by a machine vision detecting unitbased on each of the images, and a plurality of second feature points ineach of the images are generated by a deep learning detecting unit basedon each of the images. In the integrating step, the first feature pointsand the second feature points in each of the images are integrated by anintegrating unit into a plurality of integrated feature points in eachof the images. In the positioning step, a moving object is positionedaccording to the integrated feature points in each of the images.

BRIEF DESCRIPTION OF THE DRAWINGS

The disclosure can be more fully understood by reading the followingdetailed description of the embodiments, with reference made to theaccompanying drawings as follows:

FIG. 1 is a block diagram showing a feature point integrationpositioning system according to one embodiment of the presentdisclosure.

FIG. 2 is an illustration showing the first feature points in one imagegenerated by the machine vision detecting unit of the feature pointintegration positioning system of FIG. 1 .

FIG. 3 is an illustration showing the second feature points in one imagegenerated by the deep learning detecting unit of the feature pointintegration positioning system of FIG. 1 .

FIG. 4 is an illustration showing the integrated feature points in theintegrated image generated by the integrating unit of the feature pointintegration positioning system of FIG. 1 .

FIG. 5 is a relation between positioning errors and time according tothe feature point integration positioning system of FIG. 1 and acomparison example.

FIG. 6 is a block diagram showing a feature point integrationpositioning method according to another embodiment of the presentdisclosure.

FIG. 7 is a flow chart showing a previous matching step of the featurepoint integration positioning method of FIG. 6 .

DETAILED DESCRIPTION

It will be understood that when an element (or mechanism or module) isreferred to as being “disposed on”, “connected to” or “coupled to”another element, it can be directly disposed on, connected or coupled toanother element, or it can be indirectly disposed on, connected orcoupled to another element, that is, intervening elements may bepresent. In contrast, when an element is referred to as being “directlydisposed on”, “directly connected to” or “directly coupled to” anotherelement, there are no intervening elements present.

In addition, the terms first, second, third, etc. are used herein todescribe various elements or components, these elements or componentsshould not be limited by these terms. Consequently, a first element orcomponent discussed below could be termed a second element or component.Moreover, the combinations of the elements, the components, themechanisms and the modules are not well-known, ordinary or conventionalcombinations, and whether the combinations can be easily completed bythe one skilled in the art cannot be judged based on whether theelements, the components, the mechanisms or the module themselves arewell-known, ordinary or conventional.

FIG. 1 is a block diagram showing a feature point integrationpositioning system 100 according to one embodiment of the presentdisclosure. Please refer to FIG. 1 , a feature point integrationpositioning system 100 includes a moving object 110, an image inputsource 120, an analyzing module 130 and a positioning module 140. Theimage input source 120 is disposed at the moving object 110 andconfigured to shoot an environment for obtaining a sequential imagedataset. The sequential image dataset includes a plurality of images,and each of the images corresponds to each of a plurality of timepoints. The analyzing module 130 is signally connected to the imageinput source 120 to receive the sequential image dataset. The analyzingmodule 130 includes a machine vision detecting unit 131 configured togenerate a plurality of first feature points F1 (shown in FIG. 2 ) ineach of the images based on each of the images, a deep learningdetecting unit 132 configured to generate a plurality of second featurepoints F2 (shown in FIG. 3 ) in each of the images based on each of theimages, and an integrating unit 133 configured to integrate the firstfeature points F1 and the second feature points F2 in each of the imagesinto a plurality of integrated feature points F3 (shown in FIG. 4 ) ineach of the images. The positioning module 140 is signally connected tothe analyzing module 130, and the positioning module 140 receives theintegrated feature points F3 in each of the images to confirm a positionof the moving object 110 relative to the environment at each of the timepoints.

Therefore, through the second feature points F2 generated by the deeplearning detecting unit 132, the insufficiency of the first featurepoints F1 can be compensated, thereby increasing the positioningaccuracy and the positioning stability. The details of the feature pointintegration positioning system 100 will be described hereinafter.

The image input source 120 can include at least one camera, and amovable object such as a car or a robot carrying the image input source120 can be defined as the moving object 110. During the exercisingprocess of the moving object 110, the image input source 120 can shootan image series in a time series; in other words, the image input source120 shoots one image of the environment at a first time point, shootsanother image of the environment at a second time point, and keepsshooting to generate images to form the sequential image dataset.

When the analyzing module 130 receives the sequential image dataset, theimages thereof can be analyzed in real-time, the images can besimultaneously or sequentially analyzed by the machine vision detectingunit 131 and the deep learning detecting unit 132, and the first featurepoints F1 and the second feature points F2 can be generated therefrom,respectively. Please be noted that, the so-called feature points in thepresent disclosure can indicate the points in an image where the grayvalues vary obviously, or the points in an image where the outlinecurvatures of the items thereof are large, and the definition of thefeature points is well known in the technical field and will not bementioned again. In addition, in the present disclosure, for thesituation that there is no need to point out the first feature points,the second feature points or the integrated feature points, the term“feature points” will be used.

FIG. 2 is an illustration showing the first feature points F1 in oneimage generated by the machine vision detecting unit 131 of the featurepoint integration positioning system 100 of FIG. 1 , only two of thefirst feature points F1 are labeled, and the present disclosure is notlimited thereto. Please refer to FIG. 2 with reference to FIG. 1 , themachine vision detecting unit 131 can use, but not limited to,conventional feature points obtaining algorithms such as an ORB(Oriented FAST and Rotated BRIEF) algorithm or a SIFT (Scale-InvariantFeature Transform) algorithm to obtain the first feature points F1 ineach of the images. As shown in FIG. 2 , the machine vision detectingunit 131 can identify each item in the image, i.e., the road markings,and the vehicles and buildings on the side of the road, to generatecorresponding first feature points F1. However, owing to the high lightcontrast, the trees in the road straight ahead cannot be identified, andthe boundaries of the buildings on the side of the road where the lightcontrast is high are also lost.

FIG. 3 is an illustration showing the second feature points F2 in oneimage generated by the deep learning detecting unit 132 of the featurepoint integration positioning system 100 of FIG. 1 , only two of thesecond feature points F2 are labeled, and the present disclosure is notlimited thereto. Please refer to FIG. 3 with references to FIGS. 1 and 2, the deep learning detecting unit 132 can be trained in advance, and aconstructed deep learning model can be used to identify the images. Alot of environment variation images which have large environmentvariations such as the high light contrast or the direction change canbe used as the learning sources, which can be favorable for training adeep learning model suitable for environment variations. As shown inFIG. 3 , the identified image is the same as the image of FIG. 2 , andthe deep learning detecting unit 132 can clearly identify the trees inthe road straight ahead, and the boundaries of the buildings on the sideof the road where the light contrast is high.

FIG. 4 is an illustration showing the integrated feature points F3 inthe integrated image generated by the integrating unit 133 of thefeature point integration positioning system 100 of FIG. 1 , only two ofthe integrated feature points F3 are labeled, and the present disclosureis not limited thereto. Please refer to FIG. 4 with references to FIGS.1, 2 and 3 , after the first feature points F1 and the second featurepoints F2 are generated, the first feature points F1 and the secondfeature points F2 can be integrated by the integrating unit 133.Precisely, all the second feature points F2 in the image can besuperimposed on all the first feature points F1 in the image to form theintegrated feature points F3. In other words, the integrated featurepoints F3 include all the first feature points F1 and all the secondfeature points F2; as a result, the identified result from the machinevision detecting unit 131 and the identified result from the deeplearning detecting unit 132 can be remained.

For conventional feature point obtaining methods, the identification ofthe feature points is limited in high variated environment. For example,as the front light is too strong, some feature points that can be seenby human eyes may be lost. If the light of the whole image is adjusted,the feature points that are originally identified may be lost. Hence,the present disclosure uses the machine vision detecting unit 131 andthe deep learning detecting unit 132 to identify the same image, and thedeep learning detecting unit 132 can focus on the position where themachine vision detecting unit 131 is hard to identify and to find thefirst feature points F1 so as to obtain the second feature points F2 forcompensating the insufficiency of the machine vision detecting unit 131.Consequently, the integrated feature points F3 are not affected by thelight or the environment variations and the features of each item in theimage can be completely shown. After forming the integrated featurepoints F3, the positioning module 140 can confirm the position of themoving object 110 relative to the environment based on two continuousimages to complete localization. Since the feature points in two imagescan be completely shown, the position will not be lost.

Please refer to FIG. 5 with references to FIGS. 1 to 4 , FIG. 5 is arelation between positioning errors and time according to the featurepoint integration positioning system 100 of FIG. 1 and a comparisonexample, and the comparison example is a positioning result based ononly the first feature points F1, which is used to simulate apositioning system using the conventional featuring point obtainingmethod. As shown in FIG. 5 , the positioning errors of the positioningsystem of the comparison example are larger, and the positioningstability is not enough. However, the positioning errors of theembodiment of FIG. 1 remain in a small range, and the positioningstability thereof is good.

Additionally, the feature point integration positioning system 100 canfurther include a mapping module 150 configured to construct a map ofthe environment, and can construct each article in the environment basedon the integrated feature points F3, each article corresponding to eachitem of the image.

FIG. 6 is a block diagram showing a feature point integrationpositioning method 200 according to another embodiment of the presentdisclosure. Please refer to FIG. 6 with references to FIGS. 1 to 4 , afeature point integration positioning method 200 includes a shootingstep 210, an analyzing step 220, an integrating step 230 and apositioning step 240, and the details of the feature point integrationpositioning method 200 will be illustrated in association with thefeature point integration positioning system 100 hereinafter.

In the shooting step 210, an environment is shot by an image inputsource 120 to obtain a sequential image dataset, the sequential imagedataset includes a plurality of images, and each of the imagescorresponds to each of a plurality of time points.

In the analyzing step 220, a plurality of first feature points F1 ineach of the images are generated by a machine vision detecting unit 131based on each of the images, and a plurality of second feature points F2in each of the images are generated by a deep learning detecting unit132 based on each of the images.

In the integrating step 230, the first feature points F1 and the secondfeature points F2 in each of the images are integrated by an integratingunit 133 into a plurality of integrated feature points F3 in each of theimages.

In the positioning step 240, a moving object 110 is positioned accordingto the integrated feature points F3 in each of the images.

Therefore, the moving object 110 is allowed to move in an unknownenvironment, the shooting step 210 can be executed to shoot each imagecorresponding to each time point, and the images can be transmitted tothe machine vision detecting unit 131 and the deep learning detectingunit 132 via a wired transmission or a wireless transmission forexecuting the analyzing step 220 to generate the first feature points F1and the second feature points F2 in the same image, respectively.Subsequently, the integrated step 230 is executed, the integrating unit133 can obtain the first feature points F1 and the second feature pointsF2 via the wired transmission or the wireless transmission, and all thefirst feature points F1 and all the second feature points F2 can besuperimposed to form the integrated feature points F3 in each image.Moreover, in the integrating step 230, a spatial geometry modelconstructed by the multiple view geometry in computer vision can be usedby the integrating unit 133 to obtain a three-dimensional point groupdataset of the integrated feature points F3 in each of the images. Inthe three-dimensional point group, each feature point is obtained by thealgorithms in the machine vision detecting unit 131 and the deeplearning detecting unit 132 and includes feature descriptions such asthe position and the feature vectors. Finally, in the positioning step240, the position relation between the moving object 110 and theenvironment can be found by two continue images, and the localizationcan be done. Furthermore, in the positioning step 240, a map can beconstructed by the integrated feature points F3 in each of the images.

The feature point integration positioning method 200 can further includea previous matching step 250. The previous matching step 250 includesthe follows. The deep learning detecting unit 132 can be trained by aplurality of environment variation images to construct a deep learningmodel for the deep learning detecting unit 132. Two experimental imageswhich are arranged in time sequence are analyzed by the machine visiondetecting unit 131 to generate a plurality of previous firstexperimental feature points and a plurality of following firstexperimental feature points, respectively. The two experimental imagesare analyzed by the deep learning detecting unit 132 using the deeplearning model to generate a plurality of previous second experimentalfeature points and a plurality of following second experimental featurepoints, respectively. The previous first experimental feature points andthe previous second experimental feature points are integrated into aplurality of previous integrated experimental feature points by theintegrating unit 133, and the following first experimental featurepoints and the following second experimental feature points areintegrated into a plurality of following integrated experimental featurepoints by the integrating unit 133. The following integratedexperimental feature points and the previous integrated experimentalfeature points are matched to obtain a degree of similarity. If thedegree of similarity is equal to or larger than a threshold value, thedeep learning model is adapted by the deep learning detecting unit 132in the analyzing step 220, and if the degree of similarity is lower thanthe threshold value, the previous matching step 250 is repeated toretrain the deep learning detecting unit 132 to construct another deeplearning model for the deep learning detecting unit 132, and thefollowing integrated experimental feature points and the previousintegrated experimental feature points are updated to obtain anotherdegree of similarity. In other words, the present disclosure use theprevious matching step 250 to find the optimal deep learning model, andas the feature point integration positioning system 100 is carrier out,the deep learning detecting unit 132 can use the optimal deep learningmodel to obtain the second feature points F2, then the integratedfeature points F3 can be generated by the integrating unit 133, and nomatching is required.

FIG. 7 is a flow chart showing a previous matching step 250 of thefeature point integration positioning method 200 of FIG. 6 . Pleaserefer to FIG. 7 with references to FIGS. 1 to 6 , in the previousmatching step 250, the substep 251 can be executed to train the deeplearning detecting unit 132. The environment variation images used totrain the deep learning detecting unit 132 can include a plurality ofitems having light variations or position variations. One part of theenvironment variation images can be similar to the image of FIG. 2 ,which includes items such as the sky, the road, the trees and thebuildings. Since the light variation between the sky and the trees islarge, the boundaries of the trees atomize and are not easily detected.Another part of the environment variation images can include corners,and because the position variations thereof are large, the detectedfeature points in the previous image will disappear in the followingimage. In the abovementioned environment variation images, the deeplearning detecting unit 132 can be trained to focus on the positonswhere the machine vision detecting unit 131 cannot identify easily,thereby increasing the amount and the accuracy of the second featurepoints F2 detected in the environment having serious changes. Thetraining can focus on specific scenes but not on general features suchthat the correct and useful feature points can be found to compensatethe insufficiency of the machine vision detecting unit 131.

Subsequently, the substep 252 can be executed to obtain two experimentalimages which can be obtained by the image input source 120 in real-timeor can be obtained from the files stored in the database, and thepresent disclosure is not limited thereto. In the substep 253, themachine vision detecting unit 131 can analyze the two experimentalimages to generate the previous first experimental feature points andthe following first experimental feature points. In the substep 254, thedeep learning detecting unit 132 can analyze the two experimental imagesto generate the previous second experimental feature points and thefollowing second experimental feature points. The substep 255 can bethen entered to allow the integrating unit 133 to generate the previousintegrated experimental feature points and the following integratedexperimental feature points. Please be noted that the substep 253 andthe substep 254 can be executed simultaneously, or the previous firstexperimental feature points, the previous second experimental featurepoints and the previous integrated experimental feature points can begenerated prior to the following first experimental feature points, thefollowing second experimental feature points and the followingintegrated experimental feature points, and the present disclosure isnot limited to. The previous first experimental feature points and thefollowing first experimental feature points can be deemed as the firstfeature points F1 when the feature point integration positioning system100 is carried out, the previous second experimental feature points andthe following second experimental feature points can be deemed as thesecond feature points F2 when the feature point integration positioningsystem 100 is carried out, and the previous integrated experimentalfeature points and the following integrated experimental feature pointscan be deemed as the integrated feature points F3 when the feature pointintegration positioning system 100 is carried out. The feature pointsobtaining method and the integrating method are the same, and only thenames are different.

Moreover, in the substep 256, the matching is executed. As matching thefollowing integrated experimental feature points and the previousintegrated experimental feature points, a plurality of Euclideandistances are calculated, or a plurality of angles are calculated. Thedifference between the Euclidean distances and the variation between theangles can be used to calculate the degree of similarity. The higher thedegree of similarity is, the easier the localization is, and thestability thereof is high. The threshold value of the degree ofsimilarity can be, but not limited to, 75%.

Finally, the substep 257 is executed to check whether the degree ofsimilarity is larger than or equal to the threshold value. If yes, theprevious integrated experimental feature points and the followingintegrated experimental feature points are much similar, and the featurepoints are not easily lost, which means the deep learning model issuitable, and can be used when the feature point integration positioningsystem 100 is carried out. At this time, the substep 258 is entered tocomplete the previous matching step 250. In contrast, it has to go backto the substep 251 to retrain the deep learning detecting unit 132.

Although the present disclosure has been described in considerabledetail with reference to certain embodiments thereof, other embodimentsare possible. Therefore, the spirit and scope of the appended claimsshould not be limited to the description of the embodiments containedherein.

It will be apparent to those skilled in the art that variousmodifications and variations can be made to the structure of the presentdisclosure without departing from the scope or spirit of the disclosure.In view of the foregoing, it is intended that the present disclosurecovers modifications and variations of this disclosure provided theyfall within the scope of the following claims.

What is claimed is:
 1. A feature point integration positioning system,comprising: a moving object; an image input source disposed at themoving object and configured to shoot an environment for obtaining asequential image dataset, wherein the sequential image dataset comprisesa plurality of images, and each of the images corresponds to each of aplurality of time points; an analyzing module signally connected to theimage input source to receive the sequential image dataset, theanalyzing module comprising: a machine vision detecting unit configuredto generate a plurality of first feature points in each of the imagesbased on each of the images; a deep learning detecting unit configuredto generate a plurality of second feature points in each of the imagesbased on each of the images; and an integrating unit configured tointegrate the first feature points and the second feature points in eachof the images into a plurality of integrated feature points in each ofthe images; and a positioning module signally connected to the analyzingmodule, the positioning module receiving the integrated feature pointsin each of the images to confirm a position of the moving objectrelative to the environment at each of the time points.
 2. The featurepoint integration positioning system of claim 1, wherein the machinevision detecting unit uses an ORB algorithm or a SIFT algorithm toobtain the first feature points in each of the images.
 3. The featurepoint integration positioning system of claim 1, further comprising amapping module configured to construct a map of the environment.
 4. Thefeature point integration positioning system of claim 1, wherein a deeplearning model is constructed by previously using a plurality ofenvironment variation images to train the deep learning detecting unit,and the deep learning model is used to identify the second featurepoints.
 5. A feature point integration positioning method, comprising: ashooting step, wherein an environment is shot by an image input sourceto obtain a sequential image dataset, the sequential image datasetcomprises a plurality of images, and each of the images corresponds toeach of a plurality of time points; an analyzing step, wherein aplurality of first feature points in each of the images are generated bya machine vision detecting unit based on each of the images, and aplurality of second feature points in each of the images are generatedby a deep learning detecting unit based on each of the images; anintegrating step, wherein the first feature points and the secondfeature points in each of the images are integrated by an integratingunit into a plurality of integrated feature points in each of theimages; and a positioning step, wherein a moving object is positionedaccording to the integrated feature points in each of the images.
 6. Thefeature point integration positioning method of claim 5, wherein, in theintegrating step, a spatial geometry model is used by the integratingunit to obtain a three-dimensional point group dataset of the integratedfeature points in each of the images.
 7. The feature point integrationpositioning method of claim 5, wherein, in the positioning step, a mapis constructed by the integrated feature points in each of the images.8. The feature point integration positioning method of claim 5, furthercomprising a previous matching step, the previous matching stepcomprising: the deep learning detecting unit being trained by aplurality of environment variation images to construct a deep learningmodel for the deep learning detecting unit; two experimental imageswhich are arranged in time sequence being analyzed by the machine visiondetecting unit to generate a plurality of previous first experimentalfeature points and a plurality of following first experimental featurepoints, respectively, the two experimental images being analyzed by thedeep learning detecting unit using the deep learning model to generate aplurality of previous second experimental feature points and a pluralityof following second experimental feature points, respectively; theprevious first experimental feature points and the previous secondexperimental feature points being integrated into a plurality ofprevious integrated experimental feature points by the integrating unit,the following first experimental feature points and the following secondexperimental feature points being integrated into a plurality offollowing integrated experimental feature points by the integratingunit; and the following integrated experimental feature points and theprevious integrated experimental feature points being matched to obtaina degree of similarity; wherein, if the degree of similarity is equal toor larger than a threshold value, the deep learning model is adapted bythe deep learning detecting unit in the analyzing step, and if thedegree of similarity is lower than the threshold value, the previousmatching step is repeated to retrain the deep learning detecting unit toconstruct another deep learning model for the deep learning detectingunit, and the following integrated experimental feature points and theprevious integrated experimental feature points are updated to obtainanother degree of similarity.
 9. The feature point integrationpositioning method of claim 8, wherein, as matching the followingintegrated experimental feature points and the previous integratedexperimental feature points, a plurality of Euclidean distances arecalculated.
 10. The feature point integration positioning method ofclaim 8, wherein, as matching the following integrated experimentalfeature points and the previous integrated experimental feature points,a plurality of angles are calculated.
 11. The feature point integrationpositioning method of claim 8, wherein a plurality of items in each ofthe environment variation images have light variations or positionvariations.